Use projected token volume for hostNetwork pods. #428

siyanshen · 2025-01-14T18:55:22Z

Why we need this change

Before this change, GCSFuse CSI driver does not support pods with hostNetwork:true. This is because the gcsfuse process runs as part of a sidecar container that is injected into the user pod. GCSFuse uses the ADC workflow to fetch the necessary token to access a GCS bucket. However, with hostNetwork enabled, GKE metadata server cannot intercept the token requests for GET /computeMetadata/v1/instance/service-accounts/default/token endpoint.

What is in this change

For HostNetwork=true user pods, GCSFuse CSI webhook injects a projected SA token volume to the user pod.
gke-gcsfuse-sidecar container prepares a unix domain socket and starts a handler to serve requests on this token; invoke gcsfuse with a config option to point to the socket gcs-auth:token-url:<path to the token>
Token request handler in the gke-gcsfuse-sidecar handles token request from GCSFuse.

Local set up and testing

Test cases covered

pod with hostNetwork=true: can I/O GCS bucket.
pod with hostNetwork=false: is not affected and can I/O GCS bucket.
pod with hostNetwork=true, with GCS bucket A mounted as ephemeral storage and Gcs bucket B mounted as PV: can access bucket A & B without issue.

Setup

Create a cluster with workload identity pool enabled & GCSFuse disabled.
Grant GCS permissions to an SA

kubectl create namespace <ns>
kubectl create serviceaccount <test-ksa-ns> \
    --namespace <ns>

gcloud storage buckets add-iam-policy-binding gs://<bucket> \
    --member "principal://iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<project-id>.svc.id.goog/subject/ns/<ns>/sa/<test-ksa-ns>" \
    --role "roles/storage.objectUser"

Build and install your GCSFuse image

make build-image-and-push-multi-arch REGISTRY=<your-registry> STAGINGVERSION=<your-dev-tag>
make install REGISTRY=<your-registry> PROJECT=<project-id> STAGINGVERSION=<your-dev-tag>

Create a user pod with hostNetwork=true, GCS bucket mounted as a volume, and service account set as the one you created in step 2. Check I/O to your bucket from a container in your pod.

deploy/base/webhook/deployment.yaml

cmd/sidecar_mounter/main.go

pkg/sidecar_mounter/sidecar_mounter_config.go

cmd/sidecar_mounter/main.go

cmd/webhook/main.go

pkg/cloud_provider/auth/token_sources.go

pkg/sidecar_mounter/sidecar_mounter.go

pkg/webhook/mutatingwebhook.go

pkg/webhook/sidecar_spec.go

pkg/sidecar_mounter/sidecar_mounter.go

pkg/sidecar_mounter/token_manager.go

cmd/webhook/main.go

pkg/sidecar_mounter/sidecar_mounter.go

pkg/sidecar_mounter/token_source.go

pkg/sidecar_mounter/sidecar_mounter.go

pkg/sidecar_mounter/sidecar_mounter_config.go

pkg/sidecar_mounter/sidecar_mounter.go

pkg/sidecar_mounter/sidecar_mounter_config_test.go

cmd/sidecar_mounter/main.go

hime · 2025-01-31T23:33:02Z

cmd/sidecar_mounter/main.go

@@ -34,6 +34,7 @@ import (
 )

 var (
+	hostNetwork    = flag.Bool("host-network", false, "pod hostNetwork configuration")


If this goes inside mounter.Mount, do we need this flag? I think we pass (or can pass) hostNetwork flag through mountConfig.

This can simplify the sidecar injection logic too.

We still need this to pass to mount config

You are using the new HostNetwork flag to:

set mc.HostNetwork

setup gcs-auth:token-url:...

To avoid creating flags, we can determine if the Pod supports this feature on gcs-fuse-csi-driver container by checking if hostNetwork is enabled on the pod and the volume we need is injected. We have a pod informer that has that information.

gcs-fuse-csi-driver/pkg/csi_driver/node.go

Line 137 in e5d8871

pod, err := s.k8sClients.GetPod(vc[VolumeContextKeyPodNamespace], vc[VolumeContextKeyPodName])

We can then pass the mountConfig from gcs-fuse-csi-driver -> gke-gcsfuse-sidecar

driver sending mc to sidecar:

gcs-fuse-csi-driver/pkg/csi_mounter/csi_mounter.go

Line 86 in e5d8871

mc := sidecarmounter.MountConfig{

sidecar receiving mc from driver:

gcs-fuse-csi-driver/pkg/sidecar_mounter/sidecar_mounter_config.go

Line 129 in e5d8871

if err := json.Unmarshal(msg, &mc); err != nil {

After that, we can process mc to set the configmap with the uds path

gcs-fuse-csi-driver/pkg/sidecar_mounter/sidecar_mounter_config.go

Line 151 in e5d8871

func (mc *MountConfig) prepareMountArgs() {

Hey Jaime, thanks for the suggestion. Somehow the resulting pod of k8sclients.GetPod(...) does not persist the HostNetwork value. It is "false" despite my test pod setting. It might be worthwhile to derive the HostNetwork from pod.Annotations["kubectl.kubernetes.io/last-applied-configuration"], but it might be very messy.

This is the last-applied annotation.
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"gke-gcsfuse/volumes":"true"},"name":"test-pv-pod-hnw","namespace":"ns1"},"spec":{"containers":[{"command":["sleep","3600"],"image":"busybox","name":"busybox","volumeMounts":[{"mountPath":"/data","name":"gcp-cloud-storage-pvc"},{"mountPath":"/dataEph","name":"gcs-fuse-csi-ephemeral"}]}],"hostNetwork":true,"serviceAccountName":"test-ksa-ns1","volumes":[{"name":"gcp-cloud-storage-pvc","persistentVolumeClaim":{"claimName":"gcp-cloud-storage-csi-static-pvc"}},{"csi":{"driver":"gcsfuse.csi.storage.gke.io","volumeAttributes":{"bucketName":"test-wi-host-network-2","mountOptions":"implicit-dirs"}},"name":"gcs-fuse-csi-ephemeral"}]}}

So far the cleanest and safest solution is passing the bool flag from webhook. What do you think? Let me know.

Synced with Jaime offline. PodSpec was filtered in this PR: #413

This approach proves to be unfeasible even after adding back hostNetwork in podSpec. In csi driver, the msg is sent to the listner in the csi path, that looks like "/var/lib/kubelet/pods/c9a5236b-cf41-49c0-bcfe-f61988440b8f/volumes/kubernetes.io~csi/gcs-fuse-csi-ephemeral/mount". The msg received in sidecar mounter is received in the path of the volume socket: "connecting to socket "/gcsfuse-tmp/.volumes/gcs-fuse-csi-ephemeral/socket". They are not the same msg.

Will stick to the original design.

@siyanshen Do we know how all the other options are making their way from the node-server to the sidecar? Regarding the different paths being used, I believe we are using a symbolic link for socket communication. Could you clarify this so going forward we follow a different approach then?

gcs-fuse-csi-driver/pkg/csi_mounter/csi_mounter.go

Lines 211 to 215 in eeeb147

// Create socket base path.

// Need to create symbolic link of emptyDirBasePath to socketBasePath,

// because the socket absolute path is longer than 104 characters,

// which will cause "bind: invalid argument" errors.

socketBasePath := util.GetSocketBasePath(target, m.fuseSocketDir)

kislaykishore · 2025-02-05T10:43:30Z

/LGTM

hime · 2025-02-06T16:06:19Z

lgtm

siyanshen force-pushed the hostnetwork branch 7 times, most recently from 3b2b99f to 557c193 Compare January 15, 2025 23:56

siyanshen requested review from hime, saikat-royc, mattcary, msau42, kislaykishore and Tulsishah and removed request for saikat-royc January 16, 2025 00:32

mattcary reviewed Jan 22, 2025

View reviewed changes

deploy/base/webhook/deployment.yaml Outdated Show resolved Hide resolved

cmd/sidecar_mounter/main.go Outdated Show resolved Hide resolved

pkg/sidecar_mounter/sidecar_mounter_config.go Outdated Show resolved Hide resolved

cmd/sidecar_mounter/main.go Outdated Show resolved Hide resolved

siyanshen force-pushed the hostnetwork branch 2 times, most recently from 9afba55 to 2f4c6e4 Compare January 23, 2025 00:47

siyanshen removed the request for review from msau42 January 27, 2025 17:45